Longitudinal measurement invariance and psychometric properties of the Patient Health Questionnaire-Four in China

Background Depression and anxiety symptoms among medical students are often a concern. The Patient Health Questionnaire-Four (PHQ-4), an important tool for depression and anxiety screening, is commonly used and easy to administer. This study aimed to assess and update the longitudinal measurement invariance and psychometric properties of the simplified Chinese version. Methods A three-wave longitudinal survey was conducted among healthcare students using the PHQ-4. Structural validity was based on one-factor, two-factor, and second-order factor models, construct validity was based on the Self-Rated Health Questionnaire (SRHQ), Sleep Quality Questionnaire (SQQ), and Rosenberg Self-Esteem Scale (RSES), and longitudinal measurement invariance (LMI), internal consistency, and test–retest reliability were based on structural consistency across three time points. Results The results of the confirmatory factor analysis indicated that two-factor model was the best fit, and LMI was supported at three time points. Inter-factor, factor-total, and construct validity correlations of the PHQ-4 were acceptable. Additionally, Cronbach’s alpha, McDonald’s omega, and the intraclass correlation coefficient demonstrated acceptable/moderate to excellent reliability of the PHQ-4. Conclusions This study adds new longitudinal evidence that the Chinese version of the PHQ-4 has promising LMI and psychometric properties. Such data lends confidence to the routine and the expanded use of the PHQ-4 for routine screening of depression and anxiety in Chinese healthcare students. Supplementary Information The online version contains supplementary material available at 10.1186/s12888-024-05873-2.


Background
The most common mental disorders in both clinical patients and the general population are depression and anxiety, which often co-occur [1][2][3][4][5].Depression and anxiety account for more than half of mental disorders worldwide [6,7].Well known is that the co-occurrence of depression and anxiety is associated with significant disability and symptom severity, such as low back pain, poor social functioning, and multiple sclerosis [8][9][10][11][12].
Depression and anxiety usually first appear in adolescence, some of the symptoms may be acute; however, depression and anxiety can both also be chronic in nature, resulting in a huge public health burden [13][14][15].Substantial existing evidence indicates a high prevalence of depression and anxiety among healthcare students, with overall levels of psychological distress consistently higher than the general population and peers [16][17][18][19][20][21][22].The negative impact of psychological distress is far-reaching, which may adversely affect academic performance, decrease empathy, and elevate burnout in healthcare careers [23][24][25].
Most importantly, it remains to be seen whether the PHQ-4 displays longitudinal measurement invariance (LMI) [39].Of the existing evidence, only the Greek version of the PHQ-4 examined repeated surveys to assess its test-retest reliability, yet the LMI was not assessed [34].As an ultra-short instrument to screen for depression and anxiety, and track changes in these symptoms, LMI is essential to demonstrate that the construct has the same meaning across repeated assessments [27,34,49].Given the specific nature of depression and anxiety, which can occur acutely (e.g., 7 to 21 days) and chronically (e.g., weeks to years), both intervals of short-term and relatively long are worth exploring [11,[50][51][52][53][54].
The purpose of the current study was to address the following questions: 1) is the factor structure of the simplified Chinese PHQ-4 stable as a two-factor model; and 2) would the simplified Chinese PHQ-4 demonstrate adequate LMI across both short-and long-term intervals?Longitudinal measurement invariance and adequate psychometric properties for the PHQ-4 would support continued and future routine and general screening with this tool [11,26,39,51,52].Early identification and targeted prevention programs could help to prevent episodes of depression and anxiety in healthcare students [13,55,56].

Study design and participants
A three-wave longitudinal survey was conducted from December 2020 to April 2021 in Hangzhou, China.All healthcare students freely consented to answer the questionnaires.The study was approved by the Institutional Review Board of Hangzhou Normal University Division of Health Sciences, China.All procedures followed the relevant ethical tenets of the Declaration of Helsinki [57].
Healthcare students enrolled in medical courses were recruited based on the following inclusion criteria: 1) aged 17-24 years old as undergraduates; 2) not diagnosed with mental disorders; and 3) volunteered to participate in the survey.Excluded participants were primarily: 1) international exchange students who did not fully understand Chinese; and 2) on long-term leave (i.e., ≥ 3 months) for medical internship or suspension.Surveys were administered three times as baseline (T1), one-week follow-up (T2), and 15-week follow-up (T3) to allow for analyses across intervals that mimic real-world need [51-54, 58, 59].
A total of 637, 616, and 540 participants completed questionnaires at baseline, one-week follow-up, and 15-week follow-up timepoints, respectively.A total of 512 paper-pencil questionnaires were considered valid after matching.The final sample size met basic sample size considerations, which included the following: 1) the sample size should be at least 10 times the number of items in the scale; and 2) the sample size should be higher than 500 considering the estimated ratio of items to factors in the study is 2 [60,61].

Patient Health Questionnaire (Chinese Version)
The simplified Chinese version of the 4-item Patient Health Questionnaire (PHQ-4-SC, retrieved from: https:// www.phqsc reene rs.com; accessed on 29 August 2019) was the focus of the present study [27,30,62].The PHQ-4 consists of two core criteria for depression and another two for anxiety syndrome.Participants respond to the core prompt: "In the past 2 weeks, how often have the following problems bothered you?", all items (e.g., "Feeling nervous, anxious or on edge") are scored on a four-point scale marked with 0 ("not at all"), 1 ("several days"), 2 ("more than half the days"), and 3 ("nearly every day").A higher score on the PHQ-4-SC indicates poorer mental health, with total scores ranging from 0 to 12.

Self-Rated Health Questionnaire (Chinese Version)
The Self-Rated Health Questionnaire (SRHQ) consists of two items measuring physical health and mental health, respectively [49,63,64].Individuals self-report their perceived health status on a five-point Likert scale with response categories of "excellent = 1, good = 2, average = 3, poor = 4, and extremely poor = 5".The higher the total score on the SRHQ (2 to 10), the better the self-perceived health.The Cronbach's alpha of the SRHQ were 0.686, 0.672, and 0.750 at baseline, 1-week follow-up, and 15-week follow-up respectively for the current study.

Rosenberg Self-Esteem Scale (Chinese Version)
The Chinese version of the Rosenberg Self-Esteem Scale (RSES-C), one of the most widely used self-esteem instruments in the world, has two core dimensions: 1) positively worded items are scored from 1 (strongly agree) to 4 (strongly disagree); and 2) negatively worded items are reversed scored, from 1 (strongly disagree) to 4 (strongly agree) [68].After reversing the item scores, the total score ranges from 10 to 40 with higher scores representing higher self-esteem; and the RSES-C demonstrated sound measurement properties [69].

Structural validity
Confirmatory factor analysis (CFA) was first applied to determine whether the two-factor structure is consistent with the original design.The one-factor model (i.e., 4 items loaded on a general factor: psychological distress/functioning) and the two-order factor model (i.e., 2 items loaded on a depression factor and the other 2 items loaded on an anxiety factor) were selected as competing factor structures.An illustration can be found in Figure S1 of Supplementary Material.Weighted least squares mean-and variance-adjusted (WLSMV) estimation was used in all CFA analyses taking into account the ordinal nature of the item scores [77][78][79].All of the listed competing structures of the PHQ-4-SC were examined by the CFI, TLI, and RMSEA [71,[80][81][82][83].The goodness-offit (GOF) of the PHQ-4-SC was determined by thresholds (CFI ≥ 0.900, TLI ≥ 0.900, and RMSEA ≤ 0.080), the model could be considered the least suitable [83].The model with the relatively best GOF performances was selected for all subsequent analyses.

Longitudinal measurement invariance
Parameters were progressively constrained to test the LMI of the chosen structural model: configural, threshold, metric, scalar, and strict models (Supplementary Material in Table S1) [49].The scaled GOF indices (CFI, TLI, and RMSEA) together with their changes (Δ) as absolute values were used to assess LMI: 1) CFI ≥ 0.900, TLI ≥ 0.900, and RMSEA ≤ 0.080 were the least required cut-offs; and 2) |ΔCFI|≤ 0.010, |ΔTLI|≤ 0.010, and |ΔRMSEA|≤ 0.015 were the least required cut-offs.Once two, one, or no GOF indices had Δs found to fall outside the cutoffs, the model judged to be unsupported (marked red), nearly supported (marked yellow), or supported (marked green), respectively [49].The chi-squared statistic (χ 2 ) and the chi-square change (Δχ 2 ) were also compared between the models as secondary indicators, as they are sensitive to the sample size.

Construct validity
Guided by the COSMIN guidelines, we made the following hypotheses regarding construct validity [84,85]: 1) The PHQ-4-SC would positively correlate (0.300-0.500) with the SRHQ, as both measure related constructs but the SRHQ tends to focus more on health conditions.2) The PHQ-4-SC would positively correlate (0.300-0.500) with the SQQ, as both measure related constructs but the SQQ tends to focus more on sleep quality.
3) The PHQ-4-SC would positively correlate (0.300-0.500) with the RSES, as both measure related constructs but the RSES tends to focus more on selfesteem.
All of the three hypotheses (75%) had to be fulfilled for sufficient construct validity.

Internal consistency and test-retest reliability
The ordinal forms of Cronbach's alpha, McDonald's omega, and their 95% confidential interval were calculated to assess the internal consistency of the measures [86,87].Internal consistency would be considered adequate if both the alpha and omega were greater than or equal to 0.700 [84,88].
In terms of test-retest reliability, we calculated the intraclass correlation coefficient (ICC) to measure stability across timepoints.An ICC would be considered poor if it was less than 0.500, moderate if it was between 0.500 and 0.750, good if it was between 0.750 and 0.900, or excellent if it was greater than 0.900 [89][90][91].The standard error of measurement (SEM) was also calculated as an additional indicator of test-retest reliability using the formula "standard deviation × sqrt (1-ICC)" [89].

Sample characteristics
A total of 512 valid participants were included in this study.The mean age of the sample is 20.219 years, and 77.0% were female.The other demographic information and the total score of the PHQ-4-SC are summarized in the Supplementary Material, Table S2.

Longitudinal measurement invariance
On the basis of the chosen two-factor model, we conducted the longitudinal CFA to test the measurement invariance of the PHQ-4-SC across time points.Using the GOF as indicators, the LMI analysis showed that all five models are fully supported as all values (CFI = 0.998-1.000,TLI = 0.998-0.999,and RMSEA = 0.017-0.024)and their changes (|ΔCFI|= 0.000-0.001,|ΔTLI|= 0.000-0.001,and |ΔRMSEA|= 0.000-0.007)fall within the cutoffs and remain in an excellent range (Table 2).

Construct validity
Figure 2 shows the inter-factor, factor-total, and construct validity correlations of the PHQ-4-SC.We found moderate to high inter−factor and factor−total correlations with values ranging from 0.393-0.903.Most of the correlations of the PHQ-4-SC and its subscales with other measures were higher than 0.300.This partially supports the three hypotheses focused on the construct validity of the PHQ-4-SC.

Internal consistency and test-retest reliability
We observed a good internal consistency of the PHQ-4-SC with Cronbach's alpha values ranging from 0.818 to 0.919 and McDonald's omega values ranging from 0.895 to 0.916 for baseline and follow-up.Similarly, most ICC values showed moderate to good test-retest reliability was shown by ICCs ranging from 0.505 to 0.717.Notably, only the ICC of the depression subscale across baseline and 15-week follow-up was outside the moderate range (ICC = 0.453).Detailed information on the internal consistency and test-retest reliability is provided in Table 3.

Overall findings
Evidence of the measurement properties of the PHQ-4-SC from the current study revealed satisfactory performance in terms of structural validity, construct validity, and internal consistency, and of great importance, longitudinal measurement invariance over time.These results demonstrated that the PHQ-4-SC is a valid, reliable, and stable measure of depression and anxiety in the sample of health students.

Structural validity
An identical two-factor structure of the PHQ-4-SC was observed, which is consistent with the original design of the PHQ and with other adaptations of the PHQ-4 [26,27,39].The two subscales made adequate overall contributions to the PHQ-4-SC and have demonstrated the potential for it to support a bifactor model [92][93][94].Therefore, screening for both depression and anxiety as a combined disorder, rather than either one or the other alone, is also advisable if the bifactor structure is identified in the future [26,27,39,50].

Longitudinal measurement invariance
The LMI, which was the core gap of the PHQ-4 in the Chinese culture, was fully supported and provided the first evidence for the longitudinal application of the PHQ-4 in China.Given that depression and anxiety covary across time points, our design of 1-week and 15-week intervals reveals the possible ability of the PHQ-4-SC to be used for both short terms and long periods [49,50,95].However, it remains unknown whether its cross-sectional measurement invariance (CMI, e.g., gender) could be supported [49].Further analysis of the CMI on the PHQ-4-SC, which is just as important as the LMI, is needed to complete the whole picture of assessing measurement invariance.

Construct validity
Construct validity was suggested by the results of correlations between the PHQ-4-SC and the other three measures: the SRHQ, SQQ, and RSES.These are analogous to some other international studies and point to the special characteristics of depression and anxiety-as a signal for psychosomatic disorders [39].However, a missing part of the construct validity is the lack of correlations with instruments measuring similar constructs (e.g., Center for Epidemiologic Studies Depression Scale).Completing the missing part of the study would be preferable to fill the gap in assessing the construct validity of the PHQ-4 applied in the Chinese population.

Internal consistency and test-retest reliability
Despite this ultra-short instrument consisting of only four items (two items for both subscales), the internal consistency was more robust than we expected [27].This may be due to the face validity of the items, which made them easy to understand in Chinese [49,64].As for the only non-ideal ICC value, we speculate that this may be due to the long-term interval of 15 weeks-this could reduce the status of the healthcare students when they repeatedly answer the same questionnaire.This phenomenon has also been observed in another similar study [96].

Strengths and limitations
Several strengths should be highlighted.First, to date, this is the first study to evaluate the LMI of the PHQ-4-SC in a sample of the Chinese population, and to assess its use across time points.Second, this is the first study to examine the measurement properties of the PHQ-4-SC with a design including both a short-term and a longterm interval.Last, the study used multiple instruments to examine its construct validity, thus providing initial data for the analysis of risk factors for mental disorders.
Nevertheless, the study also had several limitations.First, we did not assess cross-sectional measurement invariance.Future comparisons between subgroups or characteristics should be made with caution.Second, the bifactor model was not subsequently assessed to confirm the unidimensional properties of the PHQ-4-SC.Further testing of this model would be promising for the proficiency of the overall validity of the PHQ-4-SC.Last, construct validity is the lack of correlations with instruments measuring similar constructs.Researchers are more than welcome to concurrently use other similar instruments to measure depression and anxiety simultaneously.

Conclusion
The factor structure, longitudinal measurement invariance, construct validity, internal consistency, and testretest reliability of the Chinese version of the PHQ-4 were demonstrated across three waves of measurement.Such evidence lends support for the continued and expanded use of the PHQ-4 as an effective screening instrument in China.

Fig. 1 Table 2 Fig. 2
Fig. 1 Confirmatory factor analysis results of the PHQ-4 for a two-factor model The one-sided arrows represent factor loadings while the double-sided one represents the covariance between the two factors.The orange, blue, and green color represent values of time 1, 2, and 3 respectively

Table 1
Fit indices of different factor models of the PHQ-4Bold font means that this is the best-fit model Abbreviations: χ 2 Chi-square, df degrees of freedom, CFI comparative fit index, TLI Tucker-Lewis index, RMSEA root mean square error of approximation, CI confidence interval, Δ a change in χ 2 , df, CFI, TLI, and RMSEA

Table 3
Internal consistency and test-retest reliability of the PHQ-4 This table shows ordinal forms of Cronbach's alpha (α) and McDonald's omega (ω).Standard error of measurement was calculated as "SD × sqrt (1-ICC)".The McDonald's ω and the 95% confidential interval of Cronbach's α cannot be calculated due to the anxiety and depression subscales containing only 2 items Abbreviations: PHQ Patient Health Questionnaire, Anx anxiety subscale, Dep depression subscale, SQQ Sleep Quality Questionnaire, SDS sleep difficulty subscale, DSS daytime sleepiness subscale, RSES Rosenberg Self-Esteem Scale, Pos positive subscale, Neg negative subscale, ICC intraclass correlation coefficient, CI confidence interval, SEM standard error of measurement, T1 Time 1, T2 Time 2, T3 Time 3